Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Administrator, Curator, or StaffArkiver Date: Jan 7, 2014 4:05am
Forum: faqs Subject: Re: the captures of my site are so sparse

The crawl-delay of "30" means that a crawler has to wait 30 seconds before crawling a page.

Let's say it crawls you front page, from that front page it finds other pages that are linked to from the front page. A crawler then wants to crawl those pages, the pages linked to from the front page that was crawled.
But the robots.txt files says "Crawl-delay: 30" that means a crawler needs to wait 30 seconds before crawling an other page linked to from the front page.

Hope I explained it well... :)

I will make a quick copy of your whole site for the wayback machine later today, I'll keep you informed...

EDIT:
You do not need to essentially have a robots.txt page.
If you don't have a robots.txt page a crawler can just download from the site whatever it wants.

This post was modified by Arkiver on 2014-01-07 12:05:47

Terms of Use (10 Mar 2001)